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EXECUTIVE SUMMARY 


NASA has made a substantial, multimillion-dollar investment in a World Wide Web (WWW) 
presence and in developing Web resources for use by the educational community. Over a three-year 
period, the Pennsylvania State University and the NASA Dry den Flight Research Center conducted 
educational research investigating the most effective practices for using Web-based resources to enhance 
instruction in science, mathematics, technology and geography. 

Many individuals contributed to this body of research to ensure input from as many stakeholders 
as possible regarding the issue of effective Web-based or Web-enhanced instruction. The project 
team conducted two working conferences in March and September 1997, (NASA Conference 
Publication 3358 1997 and NASA CP/1997-210722), and an analysis and needs assessment in 1998 
(NASA/TP- 1998-206547). The technical paper “Web-Enhanced Instruction and Learning: Findings of a 
Short- and Long-Term Impact Study” reports the findings of a study of a reflection tool created to help 
teachers merge the vast NASA Web resources with the best teaching methodologies. NASA 
Web-Enhanced Learning Environment Strategies (WELES) were developed, tested, revised and retested 
over the life of the project, from 1997 through 2000. The unique feature of the study was the broad 
spectrum of data collected from teachers from over 23 states in the United States and abroad. 

The feedback from a variety of teachers representing different schools, grades, states, and experience 
levels helped us develop a textual and visual representation in the NASA WELES reflection tool and 
lesson planner that is easily understandable by a vast majority of K-14 educators. 

The data analysis also revealed that innovations such as NASA WELES are much more successful 
when presented as a way to help teachers reflect on methods to enhance their current teaching and 
learning, instead of as a replacement for the methods, activities, and resources they currently use in their 
classrooms. The strategies used to facilitate NASA WELES workshops also seemed to impact teachers’ 
use of NASA resources in their teaching practices. Teachers who attended NASA WELES workshops 
that extended over a long period of time (eight months) used more NASA resources than those who 
attend the short term (one day to two weeks) continuous sessions. 

Reflection tools such as NASA WELES can and do help teachers rethink the possibilities for creating 
an ideal teaching and learning environment using any of a variety of NASA Web resources to enhance 
science, mathematics, technology, and geography lessons. Operationalizing the integration of NASA 
resources into their practice through the NASA WELES lesson planner provided teachers a familiar tool, 
a lesson plan, with which to integrate NASA Web-based resources into lessons that actively involved 
students with authentic NASA content, activities, and people. This bridge between reflecting on 
innovations and operationalizing innovations into practice was successfully accomplished. Teachers 
developed a process for integrating any of thousands of NASA Web resources into their lessons rather 
than learning how to use one new resource. 

This research has great potential to transfer beyond the educational outreach for NASA. As noted in 
this paper, the WELES reflection tool has migrated into graduate-level courses for educators, 
international educational settings, and research agendas beyond those sponsored by NASA. The true 
value of NASA WELES is that it helps teachers think about what they currently do well, what they are 
trying to do, how they plan to teach, what resources are available for teaching and learning, and how 
these resources can be effectively incorporated into lessons. More importantly, this encompassing 
framework provides teachers with prompts to think about the ultimate possibilities for teaching and 
learning, the vast numbers and types of resources available, and the endless ways to combine methods of 
teaching and NASA Web resources to make instruction exciting, motivational, and relevant to students. 
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PREFACE 


“Web-Enhanced Instruction and Learning: Findings of a Short- and Long-Term Impact Study” 
represents the culmination of three years of educational research into the best practices for using World 
Wide Web (WWW) resources for classroom instruction. Under investigation were teachers’ attitudes 
about and use of Web resources, and their feedback regarding the usefulness of a tool designed to assist 
them in combining NASA (and other) Web resources and the best pedagogical methods for teaching 
science, mathematics, technology and geography. 
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ABSTRACT 


Over a three-year period, researchers and educators from the Pennsylvania State University (PSU), 
University Park, Pennsylvania, and the NASA Dryden Flight Research Center (DFRC), Edwards, 
California, worked together to analyze, develop, implement and evaluate materials and tools that enable 
teachers to use NASA Web resources effectively for teaching science, mathematics, technology and 
geography. Two conference publications and one technical paper have already been published as part of 
this educational research series on Web-based instruction and learning. This technical paper, 
“Web-Enhanced Instruction and Learning: Findings of a Short- and Long-Term Impact Study,” is the 
culminating report in this educational research series and is based on the final report submitted to NASA. 
This report describes the broad spectrum of data gathered from teachers about their experiences using 
NASA Web resources in the classroom. It also describes participating teachers’ responses and feedback 
about the use of the NASA Web-Enhanced Learning Environment Strategies reflection tool on their 
teaching practices. The reflection tool was designed to help teachers merge the vast array of NASA 
resources with the best teaching methods, taking into consideration grade levels, subject areas and 
teaching preferences. The teachers described their attitudes toward technology and innovation in the 
classroom and their experiences and perceptions as they attempted to integrate Web resources into 
science, mathematics, technology and geography instruction. 


NOMENCLATURE 


Case-based Learning 
Activity 

Change Theory 

DFRC 

Distance Education 
Class 


Workshop activity in which the teachers were given real situations relevant to 
each participant’s context. 

A study of how innovations are implemented in an organization. 

Dryden Flight Research Center, Edwards, California 

A class offered in which most, if not all, of the course work was conducted 
virtually (through the Internet and other telecommunication means). 


ID-PRISM Represents an acronym of categories to provide a framework that supports 

reflection upon Instructional Design — Possibilities, Realities, Issues, Standards, 
and Multidimensional Perspectives created during the think tank phase of the 
educational research project with PSU and DFRC. 


K-12 

K-14 

LaRC 


Kindergarten through twelfth grade 
Kindergarten through second year of college 
Langley Research Center, Hampton, Virginia 


Lesson Planner 


A structured guide for teachers to use to formally write down their plans for 
teaching. 


NASA 


National Aeronautics and Space Administration 


NASA 

Contextualized 

Workshop 

A workshop in which NASA Web-Enhanced Learning Environment Strategies 
(WELES) training was included in an existing NASA workshop. 

NASAWELES- 
specific Workshop 

A workshop whose sole purpose was to train teachers about NASA WELES. 

Network Elements 

Capabilities of the Web that facilitate communication with other individuals. In 
other words, these are the types of resources that connect people electronically 
to enable shared interactions with experts, leaders, scientists, role models, and 
collaborators via e-mail, listservs, newsgroups, chat rooms, audio conferencing, 
and video conferencing. 

Organized Sites 

Teacher resources that are generally developed with a specific purpose in mind, 
such as to entertain, present current or historical events, provide background 
information, instruct or encourage hands-on activities. Categories include 
information, databanks, interactive events, current events, historical events, 
showcases, simulations and tutorials. 

Pedagogy 

A study of how children learn; the art of teaching; best teaching methods. 

PSU 

Pennsylvania State University, University Park, Pennsylvania 

Reflection Tool 

A structured tool to help teachers to think introspectively about their teaching. 

Site Elements 

Teaching resources that represent the raw materials from the World Wide Web 
(WWW) from which lessons can be created. Site elements include numbers, 
narratives, lists of references, still images, animations, video clips, sounds and 
tools 

TAP 

Teacher Ambassador Program 

Teaching Methods 

Specific strategies and tactics that teachers use for teaching. 

Present 

Teaching method in which the teacher provides instructional information 
through lecture, presentation, demonstration, or other “telling” activities. The 
teacher is in control as he or she specifically directs those students through the 
lesson, activity or practice session. 

Guide 

Teaching method in which the teacher uses inquiry activities to help students 
discover rules and relationships of the content. The teacher’s role shifts from 
presenter to activator of conversation with students. The teacher coaches by 
using questioning, providing context, drawing upon prior and prerequisite 
knowledge, giving feedback, and providing help. 
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Active 

Learning 

Teaching method in which the student is mentally active and constructs 
meaningful and personal understanding of the content by manipulating objects 
or abstract concepts in the learning environment. Instructional activities such as 
note taking, concept mapping, graphing, and mnemonics enable students to 
mentally “play with” information. 

Collaborative 

Teaching method in which two or more peers work together on learning 
activities that encourage them to maintain agreement and eventually reach a 
shared solution. The process promotes discussion of ideas, collaborative 
argument, and interaction among participants especially when those participants 
begin the discussion with little in common. 

Problem-based 

Teaching method which starts with a presentation of a “real world” problem in 
which students are provided with instructor guidance and resource material and 
are encouraged to dive into the problem, construct an individual understanding, 
and finally find a solution. Students actively define problems and construct 
potential solution while teachers model, coach, support and make explicit 
students’ learning processes. 

Role-Play 

Teaching method in which activities require putting the students in the context 
of a problem-based story in order to learn. The students “play” authentic roles 
while investigating a situation, identifying gaps in their knowledge, researching 
information needed to solve the problem, and developing solutions. The 
teacher’s role is to coach. 

WELES 

Web-Enhanced Learning Environment Strategies (pronounced WEE-LEEs) 

Frame 

A lesson strategy in which a context for studying a particular topic is 
established. 

Inform 

A lesson strategy in which information about the content is conveyed to the 
learner during a presentation, demonstration or discussion. 

Explore 

A lesson strategy that is student-directed. Students investigate, manipulate and 
research subject areas, problems and situations. 

Try 

A lesson strategy in which the student practices, or tests, their newly acquired 
knowledge, skills or inclinations. 

WWW 

World Wide Web 


INTRODUCTION 


Over a three-year period, the Pennsylvania State University (PSU), University Park, Pennsylvania, 
and the NASA Dryden Flight Research Center (DFRC), Edwards, California, worked together to analyze, 
develop, implement and evaluate materials and tools that enable teachers to use NASA and other Web 
resources effectively for teaching science, mathematics, technology and geography. 
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The goal of the educational research project was twofold: to identify what teachers needed to 
effectively integrate NASA and other Web resources into classroom instruction, and to develop an 
effective and useful tool to help them achieve this integration. This technical publication reports the 
findings of a short- and long-term impact study on the tool we have named the NASA Web-Enhanced 
Learning Environment Strategies (WELES). 

A broad spectrum of data was collected from teachers in over 23 states and abroad. Documentation 
from face-to-face workshops, focus groups, interviews, e-mail surveys, lesson plans and evaluation forms 
was utilized to present the teachers’ experiences. Using these data sources, we asked teachers about their 
attitudes toward technology and innovations in the classroom, and their perceptions about themselves and 
their schools’ readiness to integrate technology into classroom instruction. We asked basic demographic 
questions including their grade levels, subject area, and use of computers and technology for teaching. 
We asked open-ended questions about their current technical environment, the use of Web resources in 
their classrooms, barriers to using Web resources and their ideal learning environment. We collected data 
about their visits to NASA Web sites and evaluated the types of resources they used (and did not use). 

While the percentage of respondents to our long-term impact study was less than expected, the 
information we received over that eight-week period and the summative data over the life of the project 
gave us insights about the benefits and challenges of incorporating NASA Web resources in the 
classroom. The data confirmed some aspects about Web use that we already knew while giving us new 
insights about teachers and how they use Web resources for classroom learning. 

BACKGROUND 


One strategic educational mission of the National Aeronautics and Space Administration (NASA) is 
to increase science interest of kindergarten through second year of college (K-14) students and thereby 
inspire them to pursue science careers. As part of this educational mission, NASA has made available to 
educators hundreds of thousands of NASA-mission-related Web pages. From an analysis of teacher use 
in 1997, Grabowski, McCarthy and Koszalka found that most teachers were 1) overwhelmed by the vast 
unknown of the Internet and World Wide Web (WWW) and 2) unaware of the existence of these NASA 
educational materials. 

Recognizing a significant need, the NASA Dryden Flight Research Center (DFRC), Edwards, 
California, funded a three-year educational research project through a NASA Learning Technologies 
Project to create a strategy to address these two problems. This project capitalized on the findings of a 
think tank whose purpose was to reflect on the components needed in the design of electronic classrooms 
and to conduct an analysis and needs assessment of technology use in the United States kindergarten 
through twelfth grade (K-12) educational system. The results of these activities and a series of pilot tests 
contributed to the creation of the NASA Web-Enhanced Learning Environment Strategies (NASA 
WELES) reflection tool and lesson planner. 
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Results of the Think Tank 


Purpose and Methods 

The think tank project was launched when the Education Officer at NASA DFRC challenged the 
Pennsylvania State University (PSU), University Park, Pennsylvania, team with exploring the literature, 
instructional design models and tactics, and advances in technology to reflect on and identify components 
needed for an ideal electronic classroom. This think tank was conducted as part of an advanced graduate 
level instructional design course during which one NASA DFRC education program manager, nine 
graduate students, and one PSU professor worked through the task together for 15 weeks. 

Findings 

Encouraging teachers to think about the use of innovative technologies can lead to increases in their 
skill and knowledge of incorporating technology into their teaching practices (Pultorak, 1996; Putnam, 
1991). Recent books and articles are full of definitions of the ideal electronic classroom, prescriptions for 
how to use electronic resources, and descriptions of the effects of such resources on learning, attitude, 
and changes in pedagogy. Yet very little literature focuses on the mental processes involved in designing 
these electronic classrooms that ultimately facilitate the enhancement of instruction or the types of tools 
that can help guide teachers through the thinking processes considering the possibilities, realities, and 
issues of creating electronic teaching and learning environments. These complex factors were distilled 
into five categories to help teachers do just that. 

The resulting Instructional Design — Possibilities, Realities, Issues, Standards, and Multidimensional 
Perspectives (ID-PRISM) reflection tool represents an acronym of categories to provide a framework that 
supports reflection upon those categories as they are associated with creating an electronic classroom. As 
required in effective reflection tools, ID-PRISM prompts teachers to think about strategies they currently 
use that they believe exemplify good teaching and learning. Then, they are prompted to reflect on the 
endless possibilities for creating an ideal classroom supported with electronic resources. 

Thus, teachers are tasked with considering the multidimensional nature of the educational 
environment, namely educational and instructional design principles, learner-psychological attributes, 
and environmental and technological conditions (Koszalka, Grabowski, & McCarthy, 1999). 

Results of the Analysis and Needs Assessment 
Purpose and Methods 

To address the second issue of awareness and use of NASA Web resources in the classroom, it was 
important to establish some understanding about: 

• the contemporary educational context, that is, the administrative and technology infrastructure and 
teacher characteristics 

• school content including curriculum guidance, support and resources available and existing 
NASA Web resources 

• school processes, including best teaching practices, applied contemporary learning theory and 
methods of teaching integration. 
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The needs assessment, therefore, was organized around these three themes and was conducted by a 
16-member team: three NASA educators, two faculty members, one lead research assistant, nine doctoral 
and master’s students enrolled in the instructional systems program, and one Internet industry expert. 

Findings 

The technology infrastructure was and continues to be changing dramatically, with teacher access 
growing exponentially. Teachers’ skills however, were not keeping up with the change in access. 
Training was focused on using technology, as opposed to integrating resources into lessons. Teachers 
also perceived that it takes too much time and requires major changes in teaching to use the Internet in the 
classroom. Lastly, in terms of context, we found that administrators were critical factors in the successful 
use of the Web in the school. Teachers with administrative support were more likely to adopt the Internet 
in their classroom. 

The WWW offers NASA scientists one very convenient dissemination mechanism for extending 
NASA missions into the K-14 educational community. NASA personnel have responded by publishing a 
vast amount of information on the Web. In November of 1998, there were 291,470 Web pages with the 
nasa.gov domain name in existence (Marianne McCarthy, personal communication, November 1998). 
The content of these NASA Web resources, however, was not always teacher-friendly or school-relevant. 
To enable the teacher to use the entire site, a “translation” to school use was needed. The organization of 
NASA resources on the Web suffered from the general malaise of all Internet resources; Web sites were 
not created and posted with a systematic, teacher-friendly organizational structure in mind, nor had they 
necessarily been created for that purpose. In order to find these resources, the teachers must be very 
strategic and purposeful in their search techniques. Teachers have indicated that random searching results 
in frustratingly few usable hits. 

To reach the highest proportion of the 3.2 million teachers in the U.S, the strategy that was developed 
needed to be flexible enough to accommodate the differences between, and complexities of, the 
classroom environment. When considered appropriately, many factors that appear to impede the use of 
the Internet in the classroom can actually enable the use of NASA Web resources in effective and 
innovative ways. Two such factors include the locus of access to the Web and the use of multiple 
teaching strategies. Teachers who have direct access to the Web in their classroom, in a computer lab 
setting, in the library, in the teacher's lounge, in the administrative areas of the school, or at home can all 
use NASA Web resources. Teachers who use a variety of teaching strategies including the more 
traditional teacher-centered presentation and guiding strategies and the more learner-centered strategies 
of active learning, collaborative learning, problem-based learning, and role playing can also use the vast 
resources created by NASA. 

Given that the thousands of nasa.gov Web sites that exist can be used by the teachers in multiple ways 
as noted above, the following two scenarios exemplify the two extremes of a continuum of Internet use. 
In scenario 1, the teacher directs fifth grade students in the school lab to review the information and 
complete the interactive activities on the “Build Your Own Aircraft” Web site about understanding the 
principles of flight. The entire site is used in the manner in which it was designed to engage students. In 
scenario 2, the teacher teaches a lesson on flight dynamics and uses a variety of hands-on activities to 
create flying objects and demonstrate the principles of flight. She enhances the lesson by showing the 
students, through a liquid crystal display (LCD) projector, the animation of the four forces of flight in the 
“Build Your Own Aircraft” Web site. In this way, she frames the lesson on aircraft design at the 
beginning to promote curiosity about why objects are able to fly. In the first scenario, the teacher uses a 
NASA site “as is” in a computer lab. In the second, the teacher is selective about which elements from the 
NASA site are relevant to her own goals of the lesson, and shows them to her class with her one computer 
displayed to all. 
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Conclusions 


Based on the input from this analysis, we have reached the following conclusions regarding needs of 
teachers for using the Internet in their classroom. 

First, teachers need to have administrative support for the most widely implemented use of the Web 
in the classroom. Second, teachers need a tool that would help them integrate Internet resources smoothly 
into current teaching practices. This tool needs to focus on Web integration strategies rather than on 
general technology training. 

Therefore, we concluded that although we cannot increase administrative support, we could create a 
teacher integration tool to conceptualize Web resources similar to the way teachers currently think about 
resources in general. By using this framework, teachers could be more strategic in their searching. To this 
end, Web resources were characterized into two main types: others (human) and informational resources 
available through the Internet network and Internet sites. These human and informational resources were 
then subdivided into types of each, e.g. industry experts, peers, pictures, sounds, data. 

In a similar vein, the analysis led us to identify and categorize six contemporary teaching methods 
and six Internet access configurations for teachers. The full Analysis and Needs Assessment Report is 
also available for further reference (Grabowski, McCarthy, and Koszalka, 1997). 

The Making of Web -Enhanced Learning Environment Strategies (NASA WELES) 

Based on the data from the think tank and the Analysis and Needs Assessment, the team created the 
NASA Web-Enhanced Learning Environment Strategies (NASA WELES) as an organizational 
framework to assist K-14 teachers to plan, search for, find and use NASA Web resources to enhance 
their lessons. The NASA WELES reflection tool is shown in Figure 1. 

This figure provides an overview of the conceptual interrelationship between teaching methodology 
and WWW resources. The lower portion of Figure 1 provides the teachers with a road map through the 
myriad of resources available on the Web. The upper portion identifies six methods of teaching that the 
current educational research literature cites as the most optimal teaching strategies. The resulting four 
central WELES follow a typical lesson sequence: framing a lesson, informing learners, providing for 
content exploration by the learner, and allowing learners to try out newly acquired skills, knowledge, and 
inclinations. Teachers can often see from this overview, a broader conception of how the Web can be 
used for classroom teaching. 

Figures 2 and 3 present the lesson planner, which was developed next as a practical tool for teachers 
to use when actually planning a specific lesson. The planner steps the teachers through the thinking 
process to ensure that they consider all available teaching options for using the vast resources on the 
WWW. 

It is important to note that the resulting design took into consideration our beliefs about teaching and 
the use of Web resources, our understanding of changing pedagogy, changing realities of classroom 
access, change research, and extensive feedback from teachers. 
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Figure 1. NASA WELES reflection tool. 


8 


WELES LESSON PLANNER 



Figure 2. Page 1 of the NASA WELES lesson planner. 
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Figure 3. Page 2 of the NASA WELES lesson planner. 
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Beliefs about Teaching and the Use of Web Resources 


We held the following four beliefs and assumptions related to contemporary methods of teaching 
with WWW resources. 

• By taking a broad perspective of the WWW and the types of resources it contains, the model 
would be more useful to the teacher. 

• WWW resources have the potential to be used to enhance lessons more often than as a public 
space for publishing home pages or teacher-developed lessons. 

• Teachers were already teaching using contemporary and effective methods of teaching in a 
manner that engages learners with as many resources as they can gather together, but were 
challenged by the latter. 

• WWW resources were useful for one or more lesson strategies, or for the lesson as a whole. Web 
resources should empower teachers to re-purpose sites to meet their everyday needs, rather than 
have teachers re-purpose their own lessons to conform to the Web. 

Understanding Changing Pedagogy and the Realities of the Classroom 

We felt that the model needed to include both traditional and contemporary teaching methods to 
reach the greatest number of teachers. Traditional teaching methods included the Present , Guide , and 
Active Learning methods in which the teacher is in more direct control, reflective of a behavioristic and 
cognitive pedagogy. We then included Role-play, Problem-based and Collaborative learning in which 
the teacher plays less of a direct role, as reflective of a more constructivistic pedagogy. 

While not represented in figure 2, NASA WELES also provided advice to teachers in an 
accompanying manual that took into consideration six different computer/Intemet access configurations. 
We emphasized that if the teacher only had access at home, or in an administrative area, there were also 
ways for him or her to use this resource. Primarily, the teacher uses the resource as a planning tool for a 
vast number of lesson plans and resources that he or she could print and bring into the classroom. Of 
course, for those who have lab access, one computer with or without a display panel, or multiple 
configurations directly in the classroom, there are more options for Internet integration. 

Applying Research on Change Theory 

Theories on teacher adoption of technology innovations greatly influenced the design of NASA 
WELES. Research suggests that those teachers at ease with technology quickly migrated toward more 
sophisticated and challenging lessons incorporating new technology-enhanced resources and methods 
(Geyer, 1997). Encouraging teachers to reflect on lessons, activities, and strategies they were familiar 
with and that have worked well in the past helped them adopt new technologies (Shon, 1983; Canning, 
1991). By using an existing repertoire of successful assignments, projects, and activities from their tool 
kit, teachers had enough familiarity with old procedures to ensure a successful instructional outcome 
when applying new technologies. Continuing to experiment with Web technologies in steps encouraged 
teachers to include Web resources as a regular and continuing part of instruction rather than as rare, 
anecdotal experiences. The first step is to encourage teachers to think about successful instruction, 
current resources, and how technology can be comfortably integrated into those successful lessons. 
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This process of reflection allows for the building of logic and movement toward change (Senger, 1998). 
NASA WELES was designed to introduce Web technology innovations through methods and terms with 
which the teachers were already familiar, and had experienced success. It was important that the model 
guided teachers slowly since they were more likely to incorporate innovations a little at a time until they 
became comfortable with the innovation. 

Feedback from Teachers 

Lastly and most importantly, the NASA WELES reflection tool and lesson planner were forged and 
shaped by teacher feedback over a three-year period. We offered eight NASA-sponsored workshops 
attended by 164 teachers in the United States, five PSU graduate courses attended by 93 teachers, and one 
Assumption College Thonburi-sponsored international workshop attended by 118 Thai teachers. In each 
case, we were interested in whether the tool was effective in its ability to help teachers think strategically 
about the Web and about the use of NASA Web resources in their classroom, enough so that they would 
use them in their teaching. After each session, we considered all of the feedback, and revised the model 
accordingly. 

The following reports were written which summarized the feedback we received. Their full reports 
are available in the project final report (Grabowski & Koszalka, 2000). 

1. NASA WELES Workshop Summary: NASA Langley Research Center (LaRC), Hampton, 
Virginia — July, 1997 

2. NASA WELES Workshop Summary: NASA DFRC — July/August, 1997 

3. Summary of Web-Enhanced Learning Environment Strategies Workshops — October-December, 
1998: Keppel, Altoona 1, Lewisburg, and Altoona2 

4. Web-Enhanced Learning Environment Strategies: Dole Middle School, Hawaii Workshop 
Findings 

5. Initial Summary Report of Followup Data on NASA WELES Workshops — NASA DFRC and 
Keystone 

Feedback from NASA Education and Instructional Design Leaders 

In addition to the feedback from teachers, two NASA education leaders provided comments about the 
evolving model. In each case, comments referred to the need to simplify the model and make it very 
practical in terms of what teachers were likely to use. These suggestions were collected during frequent 
meetings over the course of the three years, and incorporated as they were given. 

Eight instructional design leaders representing six universities (PSU, Florida State, Syracuse 
University, Purdue, University of Colorado at Denver, and Iowa State University) were asked to review 
the NASA WELES model. Each reviewer was asked to respond to eight questions regarding the 
appropriateness of the strategies, attributes of the Web, components of the network, components of sites, 
networks, site elements, and configured sites, six NASA WELES, and the model in general. 
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Overall, a majority of the respondents indicated that the NASA WELES categories and model in 
general are representative. The reviewers suggested several additions to the categories, such as an 
assessment, “ask-a” Web resources, and explore/research/synthesize teaching pedagogies. Two 
respondents requested clarity or definitions of some of the items, such as Active Learning; two 
respondents had indicated that there could be overlap among some of the categories. One respondent 
brought up an interesting comment about capturing traditional teaching strategies as compared to 
providing a venue for exposure to different pedagogy. 

One of the instructional design leaders was queried extensively in a face-to-face interview, and as a 
result of this discussion, we realized that model was not the appropriate term to be using, and began 
referring to this product as a reflection tool. 

Results: The Changes in NASA WELES 

The WELES reflection tool and lesson planner evolved over the three years of this project. Each 
version was enhanced to reflect new insights on learning technologies, learning theories and teaching 
practices. Feedback from leading educational and technology experts as well as teachers who participated 
in the eight teacher workshops was also used to clarify and enhance NASA WELES materials. As a 
result, there were ten versions created over the three-year period each representing revisions based on this 
feedback. 

NASA WELES began as a conceptual drawing of the attributes of the Internet, e.g., networks and 
information resources, and instructional implementation models including six methods of teaching and 
learning, e.g. inquisitory presentation, expository presentation, generative learning, collaborative 
instruction, problem-based learning, and anchored instruction. In the middle of the model was a box 
labeled Use the Web that represented the idea of teachers merging the attributes of the Internet and the 
teaching and learning strategies they use in their classrooms. As our understanding of the use of Internet 
resources for teaching and learning, practices of applying learning theory in the classroom, 
teacher-acceptable terminology, and teacher training grew we evolved the graphical representation, 
descriptions, supporting materials, and training methods. We used our evolving understanding to enhance 
NASA WELES materials to best support teacher integration of NASA Web resources into science, 
mathematics, technology, and geography lessons. The final version was dramatically different from the 
first box-and-line representation. Contemporary teaching methods were added to the top of the model 
0 Present , Guide, Active learning, Collaborative, Problem-based, and Role-play). The parts of a lesson 
{Frame, Inform, Explore, and Try) were reflected in the center. The classification of NASA Web 
resources for enhancing lessons (site elements, organized sites and network elements) was listed on the 
bottom. The last version also had a very well developed handbook to accompany the lesson planner and 
reflection tool. 


RESEARCH METHODOLOGY 

To investigate the effectiveness and impact of the NASA WELES reflection tool and lesson planner, 
we utilized several levels of evaluation: formative evaluation, short-term and long-term impact studies, 
and NASA WELES validation. We collected data about teachers' initial perceptions of the various 
models and tools immediately following the workshop. To assess short-term impact, we tracked teacher 
use of NASA WELES for an eight- week period following the workshop. Finally, during April 1999, we 
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surveyed participants who attended workshops from July 1997 through March 1999. Long-term impact, 
therefore, was defined by length of time from NASA WELES workshop participation to the mailing of 
the survey. This length of time spanned from one to 21 months: NASA DFRC — 21 months, Keystone 
Central School District — 13 months, Keppel Union School District — six months, Pennsylvania 
Series — four months, and Dole Middle School — one month. 

Research Questions 

Our research questions were divided into two areas: general and specific. In general, we were 
interested in determining the effectiveness and usefulness of the NASA WELES reflection tool. 
Specifically, we were interested in patterns of NASA Web resource use by NASA WELES trained K-12 
teachers after being exposed to the NASA WELES. 

General Research Question: Effectiveness and Usefulness of the NASA WELES 
Reflection Tool 

1. Is the NASA WELES reflection tool comprehensible, relevant, or useful as an organizing tool for 
teachers to think about incorporating NASA Web materials in their classroom? 

Specific Research Questions: Patterns of NASA Web Resource Use by NASA 
WELES-Trained K-12 Teachers 

1. Do teachers use NASA Web resources after being exposed to NASA WELES? 

2. Which school subjects do NASA WELES-trained teachers enhance with NASA Web resources 
and with which teaching methods? 

3. Which types of NASA Web resources (e.g. images, interactive events, or scientists) do NASA 
WELES-trained teachers use? 

4. Are there differences in NASA Web resource use between participants who attended the NASA 
contextualized workshops and those who attended the NASA WELES-specific ones? Specifically, 
are they different in terms of frequency of Web integration, type of NASA resources integrated, 
teaching strategies employed, or area of school content selected for NASA Web resource use? 

5. What do NASA WELES-trained teachers like best about NASA resources, and what would they 
change? 


Subjects 

Included in this study were 164 teachers and administrators who attended the eight NASA-sponsored 
workshops between July 1997 and March 1999. Each of the 164 attendees was asked to complete the 
post-workshop evaluation. 

Of those 164 teachers and administrators, 87 were asked to complete a followup log of Web use in the 
classroom. Sixteen teachers submitted logs, representing 17 percent of the population. Either during or 
after the NASA WELES workshops, these teachers submitted 107 lesson plans. 
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Of the 135 teachers attending the workshops, 94 supplied e-mail contact information. Three messages 
were returned as undeliverable. Of the 91 surveys, we received 13 responses, despite repeated requests, 
representing a 14 percent response rate as shown in table 1. 


Table 1. Number of subjects by workshop and type of participation in the study. 


Nasa-Funded Workshop 

Formative evaluation/ 

Lesson plan 

Short-term 

Long-term 


number of attendees 

creation 

followup 

followup 



FE 

STF 

responses 

responses 

1. NEWEST: NASA LaRC— July, 1997 

Yes 

Yes 


No 

No 


N = 26 





2. Teacher Ambassador Program: 

Yes 

Yes 


No 

No 

NASA DFRC— July, 1997 

N = 3 





*Teacher Ambassador program 






Credit Course follow-on: — 1998 

N = 16 


Yes 


Yes 

(unfunded extension to TAP) 





N = 4 

3. Keystone Central School District 

No 


Yes 

Yes 

Yes 

NASA WELES Workshop: Lock Haven 
Pennsylvania — March, 1998 

N= 17 



N = 8 

N = 4 

4. Keppel Union School District 

Yes 

No 

No 

Yes 

Yes 

NASA WELES Workshop: 
Lancaster, CA — October, 1998 

N= 12 



N = 2 

N = 0 

5. Altoona School District — 1: 

Yes 


Yes 

Yes 

Yes 

Altoona, Pennsylvania — December, 
1998 

oo 

<N 

II 

z 



N = 2 

N = 0 

6. Lewisburg School District: Lewisburg, 

Yes 


Yes 

Yes 

Yes 

Pennsylvania — December, 1998 

N = 19 



N = 2 

N = 0 

7. Altoona School District — 2: 

Yes 


Yes 

Yes 

Yes 

Altoona, Pennsylvania — December, 
1998 

N = 18 



N = 2 

N = 0 

8. Dole Middle School: 

Yes 


Yes 

No 

Yes 

Honolulu, HI — March, 1999 

<N 

II 

Z 




N = 5 

Total 

164 

DFRC 

= 47 

16-17% 

13-14% 



LaRC 

= 17 





PA 

= 30 





HI 

= 13 
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While data were not collected specifically for use in this NASA-funded project, feedback and input 
from non-NASA-funded events influenced the conceptual evolution of the NASA WELES handbook, 
reflection tool and lesson planner. This input was provided by 118 Thai participants in an international 
workshop, plus 93 in-service and pre-service teachers attending five offerings of “Internet in the 
Classroom,” a distance education graduate class in which NASA WELES were presented. Since 16 of 
these same teachers also participated in the NASA Dryden workshop, only 77 are included in the total 
shown in table 2. 


Table 2. Number of workshop participants in unfunded sessions. 


Unfunded workshops and courses 

Number of 
participants 

Comments 

4 sections of “Internet in the Classroom” graduate 
class: PSU — fall 1997; spring, summer, fall 1998; and 
spring 1999 

77 

Distance Education Class 
37 additional lesson plans 
submitted 

Assumption College, Thonburi Workshops: Thailand, 
March 1998 

118 

International Workshop 

Subtotal 

195 


Grand Total exposed to NASA WELES 

359 



Demographics of Teachers 

Table 3 presents a descriptive breakdown of the demographics of the 164 participants in the various 
NASA WELES workshops showing the level of representation by grade level, teaching experience, 
computer and Web use, subject areas taught, and geographic location. Of those 164 attendees, 154 could 
be classified definitely as classroom teachers. 

As can be seen from this table, there was about equal representation between elementary and middle 
school teachers, but many more than high school teachers. Subject areas spanned most school subjects 
from science to the arts. While the different levels of experience were represented, the majority of the 
teachers had more than eight years in the classroom, with only nine teachers having less than two years 
experience. The participants came from various geographical areas of the U.S. The researchers intended 
to mitigate regional biases by attempting to involve as much geographic diversity as possible. 
Twenty-three states spanning from Hawaii to Maine and Minnesota to Florida were represented in the 
study. 
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Table 3. Teacher demographics. 



Grade level 

Teaching experience 





Total 




Use 

Use 

Subject 


number 


<2 2 to 5 to 

+ 8 

computers 

Web for 

area 

States 

of teachers 

K-4 5-8 9-12 

yrs 5 yrs 8 yrs 

yrs 

for lessons 

lessons 

represented 

represented 


NASA 

LaRC 

26 

17 

9 

0 

0 

2 

3 

21 

19 

1 

Science, 

math, 

social studies, 
language arts, 
art/music, 
generalist 

FL, KY 
MA, MD, 
ME, NC, 
NY, OH, 
PA, SC, 
TN, VA 

NASA 

DFRC 

19 

7 

7 

5 

0 

6 

5 

8 

17 

17 

Science, 

math, 

technology, 
social studies, 
generalist 

AK, ID, 
MN, MT, 
ND, OR, 
SD, WA, 
WI 

CA 

Keppel 

Union 
School =12 

3 

8 

0 

2 

2 

1 

7 

11 

8 

Science, 

math, 

generalist 

CA 

PA 

82 

KCSD = 17 
Altl = 28 
Lewisbr= 19 
Alt2 = 18 

42 

27 

10 

5 

10 

13 

53 

48 

39 

Science, 

math, 

technology, 
social studies, 
language arts, 
generalist 

PA 

HI 

Dole Middle 
School = 25 

0 

11 

8 

2 

4 

5 

14 

24 

18 

Science, 

math, 

social studies, 
generalist 

HI 

Total 

69 

62 

23 

9 

24 

27 

103 

119 

83 


23 states 

164 


154 




163 







* Note: Totals do not sum to 164 because some participants did not answer all questions. 


Treatments 

Eight different in-service workshops were offered to the participants. These workshops can be 
classified into two major types: NASA Contextualized and NASA WELES- specific workshops. 


17 


In the two NASA Contextualized workshops, NASA WELES training was included in an existing 
NASA workshop, in this case, NASA Education Workshop for Elementary Science Teachers 
(NEWEST), held at NASA LaRC, and the DFRC Teacher Ambassador Program (TAP). Both of the 
NASA Contextualized workshops were held in July 1997. In the contextualized workshops, teachers 
were immersed in NASA science content, during which we added training covering the NASA WELES. 

In the six NASA WELES-specific workshops, training was held at school district locations in 
California, Pennsylvania and Hawaii with no additional NASA science- specific content presented. The 
primary focus of the WELES-specific workshops held in 1998-99 was to learn about the NASA WELES 
reflection tool and how to use it to think about integrating NASA Web materials effectively and broadly 
in the teachers’ classroom curricula. Table 4 presents the characteristics of the eight NASA WELES 
workshops. 


Table 4. Characteristics of the eight NASA WELES Workshops. 


Workshop 

Type 

Length 

Time of 
day/year 

NASA 

WELES 

version 

Compensation 

Activities 

NASA 

LaRC 

Contextualized 

12 hours/ 8 1.5 -hr 
sessions over 
two-week period 

End of day / 
summer 

1 st 

No 

Discussion of teaching methodology 
and lab activities on use of the 
Internet 

NASA 

DFRC 

Contextualized 

16 hours/ 

2 consecutive days 

All-day, 

summer 

-,nd 

No 

Discussion of teaching methodology 
and lab activities on use of the 
Internet 

PA: KCSD 

NASA WELES- 
specific 

6 hours/ 2 sessions 
1 week apart 

Evening/ 
During the 
school year 

4 th 

IU credit 

Exemplified teaching methods, 
Internet activities, search strategies 

CA: Keppel 

NASA WELES- 
specific 

8 hours/ 2 sessions 
consecutive days 

During school 
day and school 
year 

8 th 

Release time by 
substitute 

Case-based learning, 9-step lesson 
planning, NASA mission sites, 
Internet activities, search strategies 

PA: Altl 

NASA WELES- 
specific 

8 hours 

2 4-hr sessions 

Evening/ 
During school 
year 

9th 

S50 

honorarium 

Case-based learning, 9-step lesson 
planning, NASA mission sites, 
Internet activities, search strategies 

PA: Lwsbrg 

NASA WELES- 
specific 

6 hours 
1 session 

Saturday, 
during school 
year 

9 th 

$50 

honorarium 

Case-based learning, 9-step lesson 
planning, NASA mission sites, 
Internet activities, search strategies 

PA: Alt 2 

NASA WELES- 
specific 

6 hours 
1 session 

Saturday, 
during school 
year 

9 th 

$50 

honorarium 

Case-based learning, 9-step lesson 
planning, NASA mission sites, 
Internet activities, search strategies 

HI: Dole 

NASA WELES- 
specific 

8 hours 

2 consecutive days 


10 th 

No 

Case-based learning, stories, 
analogies, electronic classroom, and 
bookmarking, embedded Internet 
activities 


Procedures 

Participants in the NASA WELES workshops were identified using various means. For the 
NASA Contextualized workshops, we were invited to add the NASA WELES training to previously 
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scheduled NASA summer workshops. For the NASA WELES-specific workshops, administrators were 
contacted in Pennsylvania and California to solicit participation by the school district. Once agreeing to 
participate, the school district itself publicized the workshop and invited participation. In Hawaii, 
contacts were made with the Hawaii Space Grant Officer who, in turn, contacted one specific school. We 
were not involved in the selection of any of the specific teachers other than to request that they had or 
would have Internet access. 

NASA provided the location for offering the contextualized workshops, while the school districts 
supplied the lab and lecture areas for the NASA WELES-specific workshops. Once the teachers were 
identified and the location arranged the workshops were scheduled and offered. At the beginning of each 
workshop, we explained the purpose of our research and the procedures for collecting data, including any 
expectations for followup. Participants were asked to complete a consent form approved by the PSU 
Office of Regulatory Compliance. 

Selected pre-assessment measurement instruments appropriate to each workshop were administered 
next, followed by the instruction as described in the previous section. At the conclusion of all but one 
workshop, participants were asked to complete a post- workshop evaluation instrument to provide us with 
formative evaluation on both the NASA WELES reflection tool and workshop strategies. In five of the 
workshops, teachers were given logbooks or weekly journals and instructions on how to keep track of 
their use of the Web over the subsequent eight weeks. Teachers were asked to send these books or 
journals weekly to a central office whose staff sent them collectively to us. Receipt of the logbooks and 
journals were tallied and teachers who did not return their forms were e-mailed or eventually called. 

After the March 1998 Keystone Central School District workshop, the teachers were contacted for a 
taped, face-to-face or telephone interview. Volunteer teachers were interviewed both as a group and 
individually to gather their thoughts about the NASA WELES reflection tool, the workshop’s impact on 
their teaching with NASA Web resources, and any additional ideas they wanted to share on the NASA 
WELES materials or approach. 

In the eight-month exposure to the NASA WELES in a graduate class following the NASA Dryden 
workshop, instead of keeping track of their Web use, teachers were asked to create NASA WELES 
lessons related to the subject’s heritage. 

In April 1999, a long-term followup survey was sent via e-mail to 91 participants to assess NASA 
Web resource use. 

Lesson Plan Submissions 

During the LaRC and DFRC NASA WELES workshops, lesson plans were created by groups of 
teachers. 

The teachers from the DFRC NASA WELES workshop continued to create NASA WELES lessons 
that integrated science, mathematics, technology, geography and Native American culture during the 
eight-month distance education class. 
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Teachers from the Keystone Central School District were not required to complete lesson plans 
during the eight weeks of logging Web use in their classrooms, however, a few had submitted outlines of 
lessons with their log books. 

The California and remaining Pennsylvania teachers were asked to submit lesson plans as part of their 
weekly journal. 

The Hawaii teachers also created lesson plans during the NASA WELES workshop. 

NASA WELES Content Validation Procedures 

Acknowledged academic experts who apply theory to practice in the fields of instructional design, 
technology, technology integration, teaching methodology and K-12 education validated the NASA 
WELES reflection tool. Experts were identified from the literature based on their expertise in these areas. 
Each was contacted to enlist their participation. If they agreed, they were sent a validation packet, which 
included the validation instruments and the approved use of human subjects consent form. Once they 
completed the consent form, these experts were asked to validate the strategies, components, 
terminology, and overall framework inherent in the NASA WELES reflection tool. Each reviewer was 
provided with a copy of the NASA WELES graphic representation and the accompanying booklet 
describing the components and framework for presenting NASA WELES to teachers. They were also 
given a validation instrument that prompted “yes” and “no” responses to each section of the NASA 
WELES reflection tool. Reviewers were also asked to comment on the terminology used in the model 
and how they might modify or otherwise change the model to better represent current classroom 
pedagogy, components of the Internet, and best practices for integrating pedagogy and Internet 
technology in teaching and learning practices. 

Measurement Instruments and Other Data Sources 

Measurement instruments were administered throughout the NASA WELES project to collect data 
about the teachers who participated in the NASA WELES workshops. Pre-assessment measurement 
instruments were designed to collect baseline data on the teachers’ attitudes, preferences, classroom 
practices, and thoughts regarding the integration of Web resources into their existing teaching 
environments. Measurement instruments were also designed to collect formative feedback on the NASA 
WELES reflection tool and workshops, specifically assessing the comprehensiveness, relevance, or 
usefulness of NASA WELES as an organization and reflection tool for teachers and the effectiveness of 
the workshops in presenting the NASA WELES reflection tool. Finally, instruments were designed to 
collect data for an eight-week period immediately following the workshop, and in a one-time survey 
administered in April 1999. The purpose of the short- and long-term followup instruments was to assess 
teacher use of NASA Web resources after being exposed to NASA WELES. 

Pre-assessment Instruments 

The pre-assessment instrument included several sections: attitude survey, school readiness 
assessment, demographic and background information, and a section on teacher use of Web technology 
in the classroom. 
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Teachers’ attitudes toward a new innovation can be predictive of their choice to use or not use an 
innovation such as NASA Web resources in their classrooms. The first section of the survey administered 
to teachers at the beginning of each NASA WELES workshop was a Teachers’ Attitude Toward the Use 
of Web Resources in Their Classroom (Koszalka, 1998) instrument. The instrument consisted of several 
questions focusing on teachers’ beliefs, feelings, and intent to use Web resources in their teaching that 
culminated in an overall attitude score. Cronbach’s alpha internal reliability was found to be 0.91. 

The second section of the instrument included a series of questions on the teachers’ assessment of 
their schools’ readiness to integrate Web resources into their classrooms. The questions were drawn from 
the items in NASA Technical Publication 1998-206547, Web-Based Learning and Instruction: Analysis 
and Needs Assessment (Grabowski, McCarthy, and Koszalka, 1998). 

The next section of the survey solicited demographic and background information on the 
participating teachers. This included information on their grade level, teaching experiences, subject area, 
and use of computers and Internet technology for teaching. 

Finally, a series of open-ended questions were included that asked teachers about their current 
technology environment, uses of resources in their classrooms, barriers to using Web resources, and their 
ideal learning environment. 

During the later workshops, beginning in October 1998, a series of worksheets were used to gather 
data on methods of teaching, use of resources, Internet access, and lesson component strategies. These 
instruments were developed as informational tools for the NASA WELES handbook as well as a means 
for collecting this information during the workshop. 

Formative Evaluation Instruments 

During the workshops the teachers were asked to assess the NASA WELES reflection tool presented 
and the effectiveness of the workshop. Formative evaluation data were collected on the use of 
terminology and graphic representation of the tool and lesson planner as well as the presentations and 
activities used during the workshops and followup session. The formative evaluation instruments were 
modified as the workshops were enhanced. 

Short-term Followup Instruments 

Daily teacher logbooks and weekly journals were designed to gather data about how teachers used 
NASA Web resources in their classrooms following the workshops. The logs were slightly different 
given the different versions of NASA WELES used with each group of teachers. 

The teacher logbooks used by the Pennsylvania teachers from Keystone Central School were 
composed of both checkboxes and open-ended questions. Teachers were asked to record how many 
lessons were covered each day, how many lessons incorporated Web resources, and detailed information 
on any one lesson that used Web resources and one that did not. For each lesson the teacher checked a 
box indicating the methods of teaching used, location of computers used, who used the Web resources, 
which types of Web resources were used, how they were used, who developed the lesson, and how 
successful the lesson was. The open-ended questions asked teachers to describe lesson procedures, key 
NASA sites visited, and non-Web NASA materials that were used during the lesson. Teachers were also 
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asked to comment on the successes experienced when using Web resources, the obstacles encountered, 
and what they would do differently in the lesson. The teachers completed the logs for one practice week 
and eight additional weeks following the NASA WELES workshop. 

The weekly journals used by the California and other Pennsylvania teachers were also composed of 
both checkboxes and open-ended questions. The checkboxes solicited specific feedback on the frequency 
of using resources in each of the four components of a lesson, e.g., Frame, Inform, Explore, and Try, the 
specific types of resources used in the lesson including site elements, network elements, and organized 
sites; and the methods of teaching. The open-ended questions asked teachers to describe their biggest 
successes and challenges in the lessons. Teachers were also provided a NASA WELES lesson planner 
and asked to provide a record of one lesson each week in which they integrated NASA Web resources 
into their teaching. These teachers were also asked to complete the weekly journals for eight weeks 
following the NASA WELES workshop. 

Group and individual debriefing interview protocols were written to gather final feedback and teacher 
thoughts on the NASA WELES reflection tool, the workshop’s impact on their teaching with NASA Web 
resources, and any additional ideas teachers wanted to share on the NASA WELES materials or 
approach. 

Long-term Followup Instrument 

The long-term followup questionnaire was designed to measure whether exposure to NASA WELES 
had an effect on teachers’ use of NASA Web resources. The questionnaire consisted of six questions. 
One question was asked to determine if teachers had prior knowledge of NASA resources before 
participating in the NASA WELES workshop. The remaining questions asked about their use of NASA 
Web resources, that is, which sites and type, which subject areas, what they liked best about the sites and 
what they would change about them. 

NASA WELES Validation Instrument 

The NASA WELES validation instrument was designed to validate the strategies, components, 
terminology, and overall framework inherent in the NASA WELES reflection tool. It consisted of the 
NASA WELES graphic representation and an accompanying booklet describing the components and 
framework for presenting NASA WELES to teachers. The questions prompted “yes” and “no” responses 
to each section of the NASA WELES reflection tool and comments about the terminology used. A 
section was included for recommendations for modification or changes to the model to better represent 
current classroom pedagogy, components of the Internet, and best practices for integrating pedagogy and 
Internet technology in teaching and learning practices. 

Lesson Plan Submissions 

Another source of data regarding the use of the NASA WELES as an effective tool to assist teachers 
in their lesson planning and use of NASA Web resources were the actual lessons teachers wrote during 
and after the workshops. We felt that a teacher’s ability to create NASA Web-enhanced lesson plans was 
an indicator of the usefulness of the tool. Lesson plans were examined for their use of Web resources to 
enhance their lessons. The lesson plans that were created during the NASA WELES workshop included 
objectives, descriptions of overall student tasks, Web preparation and specific sites, descriptions of Web 
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activities assigned to students and lesson procedures. Each lesson plan incorporated NASA resources and 
conformed to one of the six methods of teaching. 

The lesson plans that were created by the eight-month graduate class following the NASA DFRC 
workshop included 13 parts. These parts included grade level, lesson title, time required to teach the 
lesson, subject area li nk s e.g., geography, Native American culture, NASA aeronautics, education 
standards, lesson purpose and objectives, activity content, preparation and procedures, materials required, 
followup activities, and NASA Web sites. Teachers also identified the type of teaching method(s) used, 
additional content required to teach the lesson, information on non-Web NASA resources incorporated 
into the lessons, and applicable worksheets and diagrams. 

Lesson outlines included in the logbooks included lesson titles, objectives, procedures, and li nk s to 
Web resources. 

The lesson plans collected from the weekly journals were written using the NASA WELES lesson 
planner and included grade level, subject area, teaching strategies, lesson objectives, links to national 
education standards, information on the NASA mission context, lesson activities by NASA WELES, and 
NASA Web resources. 

Finally, a source of many NASA WELES lesson plans was the group of teachers who participated in 
the PSU graduate-level distance education using “Internet in the Classroom.” The lesson plans included a 
lesson title, grade level, subject, objectives, links to educational standards, lesson components and 
activities, links to Web resources, and additional notes on preparing for or conducting the lesson. 


Analysis of the Data 

The purpose of the formative, short-term, and long-term evaluation, NASA WELES validation and 
the lesson plans was to determine whether we had accomplished the goal of designing an effective and 
useful tool to help K-12 teachers integrate NASA Web resources in their classrooms. Data were analyzed 
to answer one general research question about the effectiveness and usefulness of the NASA WELES 
reflection tool and five specific questions about patterns of NASA WELES-trained teacher use of NASA 
Web resources in their K-12 lessons. 

Effectiveness and Usefulness of the NASA WELES Reflection Tool 

The general research question was: 

1. Is the NASA WELES reflection tool comprehensible, relevant, or useful as an organizing tool for 
teachers to think about incorporating NASA Web materials in their classroom? 

Table 5 relates the general research question to sources of data for analysis. 


23 


Table 5. Data sources for the general research question. 


General research 
questions 

Is the NASA WELES 
reflection tool 
comprehensible, 
relevant, useful? 


Formative 

Evaluation 

End of Teacher 

session logbook 


X 


Long -term 

Short-term followup followup Validation 


Weekly 

journal 


Lesson 

plans 


X 


Interview 


E-mail 

survey 


Validation 

instrument 


X 


Patterns of NASA Web Resource Use by NASA WELES-Trained K-12 Teachers 

The specific research questions referring to the patterns of NASA Web resource use were: 

1. Do teachers use NASA Web resources after being exposed to NASA WELES? 

2. Which school subjects do NASA WELES-trained teachers enhance with NASA Web resources and 
with which teaching methods? 

3. Which types of NASA Web resources (e.g. images, interactive events, or scientists) do NASA 
WELES-trained teachers use? 

4. Are there differences in patterns of NASA Web resource use between participants who attended the 
NASA Contextualized workshops and those that attended the NASA WELES -specific ones? 
Specifically, are they different in terms of the frequency of Web integration, type of NASA resources 
integrated, teaching strategies employed, or area of school content selected for NASA Web resource use? 

5. What do NASA WELES-trained teachers like best about NASA resources, and what would they 
change? 

Table 6 relates these specific research questions to the sources of data for analysis. 
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Table 6. Data sources for each specific research question. 


Formative Long term 

Evaluation Short term followup followup Validation 


Specific research End of 

questions session 

Teacher Weekly Lesson Interview 

logbook journal plans 

E-mail 

survey 

Validation 

instrument 

Do teachers use 
NASA Web resources 
after being exposed to 
NASA WELES? 

X X 

X 


Which school 
subjects/methods do 
teachers enhance with 
NASA Web 
resources? 

XXX 

X 


Which types of NASA 
Web resources do 
teachers use? 

X X X X 

X 


Are there differences 
between teachers who 
attended workshops 
embedded in NASA 
workshops and those 
who attend NASA 
WELES -unique 
workshops? 

X X X X 

X 


What do teachers like 
best about NASA 
resources and what 
would they change? 

X 

X 



FINDINGS 

Effectiveness and Usefulness of the NASA WELES Reflection Tool 

One general question directed our research. The findings are presented by this general research 
question. 

1. Is the NASA WELES reflection tool comprehensible, relevant, or useful as an 
organizing tool for teachers to think about incorporating NASA Web materials in 
their classroom? 

During the formative evaluation of our initial version of NASA WELES, called Use the Web , 
teachers were fairly neutral and provided us with essential feedback to make the later versions much 
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more teacher- friendly. Comments on their ratings focused on the need for more computer training rather 
than information on teaching strategies, better links to curriculum, and ‘real’ applications and examples 
for teaching. The strengths of the model were the well-designed plans, organization, and its ability to 
expose teachers to the features of the Web. When asked about NASA WELES model enhancements 
teachers generally commented that it needed better graphics, the terms needed better explanations, and 
the model needed to be better related to teaching curriculums. 

The teachers indicated that the strengths of the model were being able to see a visual representation of 
the different components and how they relate, the classification of different components of the Web, and 
that the model (reflection tool) demonstrates how teaching and the Web tie together. 

Many comments focused on NASA WELES as a useful tool for “opening doors” to the use of the 
Web, providing additional ways to think about the Web, maximizing effectiveness, and providing a focal 
point to begin to enhance lessons. Several of the teachers commented that the NASA WELES material 
helped to make lesson planning more effective and provided many helpful suggestions on how to 
integrate NASA Web resources into their curriculum. 

NASA WELES terminology for Web resources was useful to the teachers. One commented that the 
categories were helpful in breaking the Web down into manageable parts and several others indicated that 
the categories were very easy to relate to, and made them more comfortable with the Web. Several had 
commented that the categories and terminology made Web searching easier and would help them make 
better use of their planning and teaching time. 

Most of the teachers commented that the list of teaching methods was appropriate and well-rounded. 
Some commented that the list of teaching methods was not new, while others commented that it was a 
refreshing review and reminder of strategies they had not used in a while. 

The teachers thought that the categories used to describe the resources available on the Web could 
easily be related to and were useful in thinking about Web resources. Several indicated that they liked the 
categories and now realized the importance of understanding the types of resources available and 
matching the types of Web resources to the types of activities they plan in their classrooms. 

Teachers indicated that the NASA WELES lesson planner was succinct and useful, especially for 
novice teachers and for helping experienced teachers organize their thoughts about using NASA Web 
resources in their lessons. 

Teacher interview comments indicated that the early version of the NASA WELES reflection tool 
provided them with Web integration ideas, guidelines that used understandable terminology, references 
that helped extend their classroom and lesson ideas, and clarifications on methods of teaching with Web 
resources. 

Overall, a majority of the respondents to the validation tool indicated that the NASA WELES 
categories and model in general are representative. All had agreed that the strategies cited in the 
model — Present , Guide, Role-play, Active learning. Collaborative, and Problem-based learning — 
applied to today’s teaching practices. However, two of the reviewers suggested adding assessment, while 
three of the reviewers suggested adding clarification and further definition of the teaching strategies. 


26 


Conclusions 

As an overall summary of the findings for the general research question, a majority of the teachers 
indicated that the language of the early versions of NASA WELES was not quite comprehensible to them 
without explanation. However, teachers indicated that the terminology and language of the model 
became more comprehensible and relevant in subsequent versions of the NASA WELES reflection tool 
as it emerged over time. 

Teachers also indicated that from the beginning the NASA WELES reflection tool was useful in 
helping them to reflect and think about a variety of uses of NASA Web resources to enhance their 
teaching and learning environments. Many comments throughout the subsequent versions indicated that 
NASA WELES helped teachers organize their thoughts about the types of Web resources available and 
their approaches to lesson planning using Web resources. 

The experts indicated that the NASA WELES reflection tool was representative of the methods of 
teaching, Web resources, and learning strategies used in the classroom. They recommended that some of 
the categories be adjusted to account for emerging pedagogy and Web resources and include further 
explanations of NASA WELES as the merging of pedagogy and Web resources. 

Overall, although there were a few teachers who questioned the relevance of the NASA WELES 
tools, a majority indicated that they provided a useful and relevant approach to harnessing the vast 
number of NASA Web resources for specific uses in the many different methods they use for teaching in 
their classrooms. 

Patterns of NASA Web Resource Use by NASA WELES-Trained K-12 Teachers 

Five specific research questions directed our research. The findings are presented by these specific 
research questions. 

1. Do teachers use NASA Web Resources after being exposed to NASA WELES? 

Forty-five of the 107 lesson plans were selected from the NASA DFRC TAP participants for analysis. 
The 16 participants who took the follow-on credited course in which they were not required to use NASA 
resources wrote these lesson plans. Teachers expected to practice the principles from the NASA WELES 
workshops, including the use of NASA resources, wrote the others. We felt that these 45 lesson plans, 
therefore, represented the least biased sample of the total lesson plans written. By analyzing these lesson 
plans, it is evident that these participants were not only aware of the existence of NASA sites, but used 
them in their lesson plans. Twenty-five, or 56 percent of these lessons submitted, included some NASA 
sites. One lesson plan was submitted that included NASA materials that were not Web-based. 

From the group and individual interviews, it was evident that this sample of teachers increased their 
use of Web resources in general. One teacher stated, “I tried to use more. The log increased my frequency 
of Web resource use. It increased the number of assignments I give that require Internet resources.” Two 
teachers indicated that they felt that use of Web resources was important because it “provides 
applications and ideas [and takes them] beyond the walls of the classroom to a world of possibilities” and 
it “increases my terminology and interest.” While NASA sites were not specifically mentioned in these 
responses, other data sources included NASA sites in the list of Web resources that teachers had used 
during this study. 
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In the long-term followup e-mail survey, eight of the 13 teachers responded that they did not know 
about NASA Web resources prior to attending the NASA WELES workshop. Five responded that they 
were aware of NASA Web resources; four indicated that they had used NASA Web resources in their 
classrooms previously. This pattern continues to be consistent with other teachers with whom we have 
contact. 

After being exposed to the NASA WELES, nine teachers indicated that they now used NASA Web 
sites. One indicated that she was on sabbatical but planned to use NASA resources when she returned to 
teaching. Only two of the teachers, not on sabbatical, did not use NASA Web sites at all. 

Conclusions 

Based on the data from all of the sources, we can conclude that teachers increase their use of Web 
resources in general after being exposed to NASA WELES. Based on specific instances included in the 
lesson plans and the e-mail survey, teachers exposed to NASA WELES are also more likely to use NASA 
Web resources in their classroom. 

Note: Due to low response rates of the e-mail surveys, conclusions from the data must be considered 
cautiously. The trends noted for those who responded were encouraging. However, those who do not use 
technology often may be less likely to respond to such surveys than people who use technology resources 
regularly. Thus, without data from a larger sample of all those involved the research findings cannot be 
generalized to the greater population of K-14 educators. 

2. Which school subjects do NASA WELES-trained teachers enhance with NASA 
Web resources and with which teaching methods? 

Eight teachers from Keystone Central School District representing 7 th grade science, 3 ld , 4 th , and 10 th 
grade mathematics, and 3 ld , 4 th , and 6 th grade generalists submitted logbook data for a nine-week period. 
Over these nine weeks, 40 instances of Web use were noted, of which eight specific NASA sites were 
used. 

An examination of the weekly journals for teaching methods revealed that the Explore strategy was 
the NASA WELES strategy used the most, with 41 instances, and Try was used the least, with 17 
instances. The Frame and Inform strategies were used a similar number of times, with 32 and 34 
instances, respectively. 

In addition, the teachers used only five of the six teaching methods. The method teachers most often 
used was Active learning, and not used at all was Role-play. Collaborative learning and Problem-based 
learning, the two most difficult teaching methodologies, were only used four and three times, 
respectively. 

Examining the lessons submitted by the NASA DFRC, NASA LaRC, Pennsylvania and Hawaii 
teachers, and additional lesson plans submitted by students in the PSU distance education courses 
exposed to NASA WELES, it is evident that Web materials in general were incorporated in seven 
different subject areas. These subject areas listed in the order of most to least number of times they were 
included: science, mathematics, social studies (culture, history, and social science combined), 
technology, geography, language arts, and art. 
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NASA DFRC lesson plans were analyzed for the number of lesson plans that used NASA Web 
resources. The four content areas with the highest percentage of NASA Web resource use include art, 
mathematics, technology and science. Overall, 44 percent of the lessons submitted included some NASA 
Web resource. 

From analyzing the lesson plans created, one can note that teachers have incorporated Web resources 
using a variety of different methods of teaching predominantly while using present strategies and active 
or generative learning. 

Many lessons incorporate Web resources for more than just searching. For example, Web resources 
are used in half of the lessons to frame activities or excite and motivate students. The Web resources are 
used to give student specific information and as sources of information for students to develop projects, 
share ideas and data with others, and manipulate concepts through simulations and tools. Most work on 
the Web is supplemented with offline activities. 

Teachers were asked in a group interview about which new strategies they had tried and would 
continue to use. Included in the methods that they mentioned were collaboration, expository 
presentations, and an interest in exploring all the strategies a little more than they had before exposure to 
the WELES reflection tool. 

In an e-mail followup survey, nine teachers indicated that they used NASA Web sites in science 
classes, two used them in mathematics classes, four in technology classes, two in a geography class, and 
one in an aviation class. One indicated that she was on sabbatical but planned to use NASA resources 
when she returned to teaching. Some teachers used NASA resources to teach multiple subjects. Two of 
the teachers, not on sabbatical, did not use NASA Web sites at all. 

Conclusions 

The data from the logbooks, weekly journals, lesson plans and e-mail followup survey all indicate 
that NASA Web resources can be and are used in at least eight content areas: science, mathematics, social 
studies, technology, geography, language arts and art. 

In terms of teaching methods and use of the various NASA WELES, data from the weekly journals 
and lesson plan analysis, indicate that all four of the NASA WELES strategies, Frame , Inform , Explore 
and Try hold possibility for being taught with NASA Web resources. Methods employed, however, 
showed a different pattern, with Active learning methods being most used with NASA Web resources. 
Collaborative learning and Problem-based, learning are being used but only minimally. The purpose of 
the reflection tool is to encourage teachers to consider using a variety of Web resources with a variety of 
teaching methods, and at the same time reach out to teachers with methods that they are already 
comfortable with. The data suggest a few adventurous teachers, but also they suggest that more 
concentrated effort be placed on ways to raise the consciousness level of teachers to the more innovative 
teaching methods with some of the more innovative Web resources. 

Interview data suggest that NASA WELES has the possibility of encouraging new methods of 
teaching, with 35 percent of those teachers indicating that they are doing something differently. 
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3. Which types of NASA Web resources (e.g. images, interactive events, or scientists) 
do NASA WELES-trained teachers use? 

By examining the teacher logbook from the Keystone Central School District, two different types of 
NASA Web resources were noted as being used: images/pictures/information and lesson plans. Five 
instances were noted for the images and three for lesson plans. These were taken from six specifically 
noted sites. By examining the weekly journals from the California and remaining Pennsylvania Schools, 
we noted 19 different types of resources in 276 instances of use over the eight- week period. Information, 
narratives, still images and animations were selected the most. Network elements were selected only 
minimally. This is consistent with the fact that teachers used collaborative teaching methods only 
minimally as well. It must be pointed out, however that it is not evident from the data that we collected if 
these were specifically NASA sites or sites in general. 

When asked in interviews about the types of Web resources that they had used during this study, the 
teachers responded with 14 types. These included Netscape, lesson plans, lists of resources, plug-ins, 
simulations, calculators, and databases, lists of other teachers, NASA sites, scholastic news site, 
mathematics and visual projects, Ask sites, and projects. While these responses did not include specific 
NASA sites, we did not ask for specific sites in this question. Other responses from other instruments 
noted many NASA sites that were used. It is highly likely, therefore, that many of these noted in the 
interviews were NASA sites. 

In an e-mail followup survey with regard to which NASA sites they were using, most participants 
provided general descriptions of the sites they have used to teach. These included: nasa.gov, SR-71, 
weather, space, atmosphere, airplane ground school, airplane construction, geology tracking, JPL, 
planets, NASA control panel, how machines work, Galileo, Mars Pathfinder and links. Some also 
indicated that they have used too many to list or many others. 

When asked which types of Web resources they were using, ten indicated that they used images, nine 
used number or picture databases, six used NASA lesson plans, nine used general NASA information 
sites, and two have communicated with NASA people. Two have been involved with NASA interactive 
student project sites, one has used NASA tutorials, and no one indicated using any other types of NASA 
Web resources. 

Conclusions 

From the data, we can conclude that NASA WELES-trained teachers use a variety of Web resources. 
At the beginning of the project, fewer numbers of types of Web resources were reported, with many more 
resources reported later on. A pattern showing little use of network Web elements was consistent even 
with the final e-mail survey. Weekly journals showed only five instances of network element use, and 
only two in the final e-mail survey. These results indicate that teachers may again gravitate to those 
resources with which they are most familiar, and not try out new opportunities to connect their classroom 
with outside others. They may also reveal a suspected increased level of complexity and probability of 
technical problems. 
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4. Are there differences in NASA Web resource use between participants who 
attended the NASA Contextualized workshops and those that attended the NASA 
WELES-specific ones? Specifically, are they different in terms of the frequency of 
Web integration, type of NASA resources integrated, teaching strategies employed, 
or area of school content selected for NASA Web resource use? 

A comparison can be made between the lesson plans from the NASA Contextualized workshop held 
at NASA DFRC and the Keystone Central School District logbooks, and the weekly journals from the 
NASA WELES-specific workshops completed by Keppel Union School District and the three other 
Pennsylvania Schools. Interview data were only collected from the NASA WELES-specific workshop. 

From the interview data representing the NASA WELES-specific workshop, 12 types of resources 
were listed along with two different teaching methods. Three respondents included four specific 
strategies, while three indicated their use of multiple strategies. 

In the e-mail followup survey, the teachers in the contextualized workshops reported using NASA 
Web resources more frequently and using more types of resources in more topics than those in the 
WELES-specific workshop. There were also some interesting patterns in the types of resources used. For 
example, many more teachers from the contextualized workshop were using the information-based 
resources, e.g. images, animation, and lesson plans rather than people resources, e.g. NASA scientists 
and experts. As compared to the WELES-specific workshop teachers, these teachers also used a variety 
of NASA sites as well as other science Web resources. Teachers commented that they have used teacher 
strategies with these Web resources that they had not regularly used in the past. One teacher who 
primarily used Present strategies stated that she began to use more Problem-based learning and 
Collaborative learning methods in her class. Another teacher who was primarily using Active learning 
methods admitted to using more Present methods because of the interesting materials on the Web and 
lack of access for all students. 

Conclusions 

An important conclusion regarding diffusion of the reflection tool is that the more contextualized the 
workshop describing NASA WELES is within a specific topic, the more likely the Web will be used to 
teach that specific topic. If one’s goal is to enhance science teaching with NASA resources, our 
recommendation is to introduce NASA WELES in an existing NASA teacher workshop. If one’s goal 
were to enhance teaching in general without regard to resources, methodology, or school subject, then a 
more generic workshop would be appropriate. A second very important conclusion, is that followup is 
important, but not in the form of a listserv. The workshop that extended over the eight-month period of 
time had a greater impact on teacher use of NASA Web resources, than those who attended a short-term 
continuous session. 

5. What do NASA WELES-trained teachers like best about NASA resources, and 
what would they change? 

When asked about what Web resources in general offered to the teachers in the classroom, teachers 
noted ten different characteristics. These included enhanced communication, lesson plans, ideas for 
lessons, alternate research resources, access to other information, updated information, a means for 
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students to reach beyond the classroom, enriched text, and opportunities to motivate students and make 
them explorers. 

In the e-mail followup survey teachers were asked specifically about NASA Web resources. To this 
question, teachers gave eight positive responses. They indicated that they thought that the NASA Web 
resources increased student interest, provided information not easily obtained otherwise, provided 
beautiful graphics, were interesting, provided cutting-edge information, were vast and informative, 
contained current science events, and were useful for teaching. One teacher stated that she was able to 
find information that all of her students (grades 2-6) could understand and that they really enjoyed the 
interactive sites. One also mentioned that she most liked bringing scientists and experts into the 
classroom through the NASA Web sites. 

Teachers indicated that to improve NASA Web resources, they would like descriptions of specific 
sites with specific addresses to get to the specific information or that there should be an easily accessible 
catalog of NASA Web resources. 

Conclusions 

Teachers like NASA Web resources. They cite motivational, content, quality, currency and 
usefulness to themselves and their students as their reasoning. 

Consistent with what we heard and continue to hear from teachers is that they would like to see an 
easy means to determine what NASA resources exist and how to access them. 

CONCLUSIONS 


The results of our three-year effort lead us to conclude that the NASA WELES lesson planner and 
reflection tool make an important contribution to helping teachers think about Web resources, especially 
NASA Web resources, and how those resources can be integrated into their classroom teaching. 

Conclusions about the WELES Reflection Tool and Lesson Planner 

Terminology and Graphic Representation 

Working with educators and teachers, as with any member of a profession, requires using the 
language of that practice. Using text and graphic representation that do not represent “teacher speak” 
affects teachers’ level of comfort with innovative ideas and new resources. For example, many of the 
NASA WELES workshop teachers understood the concepts we used when describing the Internet 
attributes and methods of teaching, however often stumbled with the terminology we used in the earlier 
versions of the NASA WELES reflection tool and lesson planner. Working with a variety of teachers, 
from different schools, grades, states, and experience levels helped us develop text and visual 
representation in the NASA WELES reflection tool and lesson planner that is easily understandable to the 
largest possible percentage of the nation’s 3.2 million practicing teachers. Thus, the text and graphic 
representations of NASA WELES are familiar to and can be easily understood by a vast majority of K-14 
educators because they helped us craft the language of the tool. 
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Enhancement Rather Than Replacement of Teaching Practices 

Teachers, especially those who have been teaching many years, are masters of their classrooms. They 
know their classroom environment, teaching preferences, and students. When helping teachers integrate 
new ideas and strategies into their teaching, such as the work done with NASA WELES, it is important to 
acknowledge that the methods, strategies, and resources they currently use in their classrooms are 
appropriate. 

Innovations such as NASA WELES are much more successful when presented as a way to help 
teachers reflect on enhancing their current teaching and learning, not as a replacement for the methods, 
activities, and resources they currently use in their classrooms. 

Seeing the Possibilities, Reflection to Practice 

Past research has demonstrated that the possibilities for technology use in the classroom extend far 
beyond the individualized drill- and-practice and entertainment scenarios of the past. Many teachers still 
hold to the ideas that the Internet is simply a giant entertainment or search tool, it is dangerous for kids, 
and it is often an unproductive class resource because of the possibilities for distraction from class work. 
Helping teachers to see the relationships between the way they teach and the many different types of 
NASA resources available, including information, interactive events, and human, helps them think about 
the possibilities for enhancing their teaching and learning environments. Giving teachers an Internet 
resource to use rather than a framework like NASA WELES is like the old proverb, giving a hungry man 
a fish to eat will feed him for a day . . . teaching him how to catch fish will feed him for a lifetime. 
Reflection tools such as NASA WELES can and do help teachers rethink the possibilities for creating an 
ideal teaching and learning environment using any of a wide variety of NASA and other Web resources 
to enhance science, mathematics, technology, and geography lessons. 

Operationalizing the integration of NASA resources into their practice through the NASA WELES 
lesson planner provided teachers a familiar tool, a lesson plan, with which to integrate NASA Web-based 
resources into lessons that actively involved students with authentic NASA content, activities, and 
people. This bridge between reflecting on innovations and operationalizing innovations into practice was 
successfully accomplished using the NASA WELES framework. Thus, exposure to the NASA WELES 
reflection tool encouraged teachers to think about the possibilities for teaching and learning while the 
NASA WELES lesson planner aided them in developing new and enhancing existing lessons that did 
indeed incorporate such resources in their practices. Teachers developed a process for integrating any of 
thousands of NASA Web resources into their lessons rather than learning how to use just one new 
resource. 

Making the Exotic Familiar and the Familiar Exotic 

Shon (1983) warns, “As practice becomes more repetitive and routine, and as knowing-in-practice 
becomes increasingly tacit and spontaneous, the practitioner may miss important opportunities to think 
about what he is doing” (p.6). Geyer (1997) found that encouraging teachers to think about lessons, 
activities, and strategies that worked well in the past helps them adopt new technologies. By using an 
existing repertoire of successful assignments, projects, and activities from their tool kit, teachers have 
enough familiarity with old procedures to ensure a successful instructional outcome using new 
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technologies. Yet, the first step is to encourage teachers to think about successful instruction and how 
technology can be comfortably integrated into those successful lessons. 


Conclusions of the Impact Study 

Teachers indicated that the NASA WELES reflection tool, from the beginning, was useful in helping 
them to reflect and think about a variety of uses of NASA Web resources to enhance their teaching and 
learning environments. Many comments throughout the subsequent versions indicated that NASA 
WELES helped teachers organize their thoughts about the types of Web resources available and their 
approaches to lesson planning using Web resources. 

Based on the data from all of the sources, we can conclude that teachers use Web resources in general 
after being exposed to NASA WELES. Based on specific instances included in the lesson plans and the 
e-mail survey, teachers exposed to NASA WELES are more likely to use NASA Web resources in their 
classroom. 

The data from the logbooks, weekly journals, lesson plans and e-mail followup survey all indicate 
that NASA Web resources can be and are used in at least eight content areas: science, mathematics, social 
studies, technology, geography, language arts and art. Examining the lessons submitted by the NASA 
DFRC, NASA LaRC, Pennsylvania and Hawaii teachers, and additional lesson plans submitted by 
students in the PSU distance education courses exposed to NASA WELES, it is evident that Web 
materials in general were incorporated in seven different subject areas. These subject areas listed in the 
order of most to least number of times they were included: science, mathematics, social studies (culture, 
history, and social science combined), technology, geography, language arts, art, and aviation. 

From the data, we can conclude that NASA WELES-trained teachers use a variety of Web resources. 
At the beginning of the project, fewer numbers of types of Web resources were reported, with many more 
resources reported later on. A pattern showing little use of network Web elements was consistent even 
with the final e-mail survey. Weekly journals showed only five instances of network use, and only two in 
the final e-mail survey. These results indicate that teachers may again gravitate to those resources with 
which they are most familiar, and not try out new opportunities to connect their classroom with outside 
others. 

If one’s goal is to enhance science teaching with NASA resources, our recommendation is to 
introduce NASA WELES in an existing NASA teacher workshop. If one’s goal were to enhance teaching 
in general without regard to resources, methodology, or school subject, then a more generic workshop 
would be appropriate. A second very important conclusion, is that followup is important, but not in the 
form of a listserv. The workshop that extended over the eight-month period of time had a greater impact 
on teacher use of NASA Web resources, than those who attended a short term continuous session. 

Next, teachers like NASA Web resources. They cite motivational, content, quality, currency and 
usefulness to themselves and their students as their reasoning. 

Finally, and consistent with what we heard and continue to hear from teachers is that they would like 
to see an easy means to determine what NASA resources exist and how to access them. 
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In conclusion, NASA WELES helps teachers think about what they currently do well, what they are 
trying to do, how they plan to teach, what resources are available for teaching and learning, and how 
these resources can be effectively incorporated into lessons. More importantly, this encompassing 
framework provides teachers with prompts to think about the ultimate possibilities for teaching and 
learning, vast numbers and types of resources available, and endless number of ways to combine methods 
of teaching and NASA Web resources to make instruction exciting, motivational, and relevant to 
students. Teachers are prompted to think about integrating new and different types of resources into their 
familiar practices while considering how to enhance their existing practices whether they are in the form 
of projecting pictures from the Internet, encouraging student exploration of a site, or collaborating with 
NASA scientists. 
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PRESENTATIONS AND PUBLICATIONS 


Over the course of the three years of funding, the PSU/NASA staff presented at conferences, 
published documents and papers, and conducted workshops and courses about NASA WELES. Eighteen 
professional presentations were made at local, national, and international organizations. The project 
produced 22 publications, including two NASA published documents, eight project-specific documents, 
two Conference Proceedings, one refereed journal article, and six project research reports. Eight NASA 
contract-funded workshops and seven non-NASA-contracted graduate classes, and one conference were 
held during which NASA WELES was taught. Finally, non-project writers highlighted the NASA work 
at PSU by publishing four other articles. 
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PROFESSIONAL PRESENTATIONS 


Date 

Type 

Conference 

Title 


Feb-98 

Presentation 

AECT/St. Louis 

Using effective Web-based materials in the 
classroom 

Grabowski, Koszalka, 
and McCarthy 

Feb-98 

Presentation 

AECT/St. Louis 

Through the ID-PRISM: Reflecting on 
instructional design for electronic classrooms 

Grabowski, McCarthy, 
Koszalka, and Hernandez 

Mar-98 

Presentation 

Thailand 

World WideWeb-Enhanced Learning: An 
Exciting and a Challenging Way of Instruction 

Grabowski 

Apr-98 

Presentation 

Penn State Univ. 

Web-Enhanced Learning Environment Strategies: 
The WELES applied to Microscopy! 

Grabowski 

Apr-98 

Presentation 

Penn State Univ. 

Web-Enhanced Learning Environment Strategies: 
More Than Just Web-Page Development 

Grabowski 

Jun-98 

Presentation 

LTP 

Web-Enhanced Learning Environment Strategies 
*WELES* update 

McCarthy, Grabowski, 
and Koszalka 

Jul-98 

Presentation 

INET/ Geneva, 
Switzerland 

Web-Based Instruction and Learning: Analysis 
and Needs Assessment Summary 

McCarthy, Grabowski, 
and Koszalka 

Jul-98 

Presentation 

INET/ Geneva, 
Switzerland 

Web-Enhanced Learning Environment Strategies 
for Classroom Teachers 

Grabowski, Koszalka, 
and McCarthy 

Aug-98 

Presentation 

Syracuse Univ. 

Entering the New Millennium doing “WELES !” 

Grabowski 

Feb-99 

Presentation 

AECT/Houston 

Guidelines for sharing lesson plans over the Web 

Moore, Koszalka, and 
Breman 

Feb-99 

Presentation 

AECT/Houston 

Web-Based Instruction and Learning: Analysis 
and Needs Assessment 

McCarthy, Grabowski, 
and Koszalka 

Feb-99 

Presentation 

CoSN/ 

Washington D.C. 

ID-PRISM: Reflecting on the Electronic 
Classroom 

Koszalka, Grabowski, 
and McCarthy 

Feb-99 

Presentation 

CoSN/ 

Washington D.C. 

Web-based Instruction & Learning: Analysis and 
Needs Assessment 

McCarthy, Grabowski, 
and Koszalka 

Feb-99 

Presentation 

CoSN/ 

Washington D.C. 

Web-Enhanced Learning Environment Strategies: 
Integrating Web resources into your classroom 

Grabowski, Koszalka, 
and McCarthy 

May-99 

Presentation 

LTP 

Research Findings of the NASA Dryden Flight 
Research Center Learning Technologies Project 

McCarthy and Grabowski 

Nov-99 

Presentation 

P-AECT/ 
Harrisburg, PA 

What do the kids think? A study of resource use in 
middle school classrooms 

Koszalka 

Feb-00 

Presentation 

AECT/ 

Long Beach, CA 

Web-Enhanced Learning Environment Strategies: 
Integrating NASA Web resources into science 
instruction 

Koszalka, Grabowski, 
and McCarthy 

Feb-00 

Presentation 

AECT/ 

Long Beach, CA 

The Validation of a Measurement Instrument: 
Teachers’ Attitudes Towards The Use of Web 
Resources in the Classroom, Across Two 
Cultures 

Koszalka, Prichavudhi, 
and Grabowski 


AECT = Association of Educational Communication Technology; CoSN = Consortium of School Networking; 
IADIS = International Association for the Development of Information Society; INET = Internet Society; 

LTP = NASA Learning Technologies Project; P-AECT = Pennsylvania AECT 
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PROFESSIONAL PRESENTATIONS, CONTINUED 


Published Refereed Conference Proceedings 

Date 

Type 

Conference 

Title 


2000 

2000 

Conference 

Proceedings 

AECT 

(Sponsored by 
Research and 
Theory) 

Web-Enhanced Learning Environment Strategies: 
Integrating NASA Web Resources into Science 
Instruction 

Koszalka, T., 
Grabowski, B., and 
McCarthy, M. 

2001 

2000 Denver 
Conference 
Proceedings- 
Research and 
Theory edition 

AECT 

The Relationship between the Types of Resources 
Used in Science Classrooms and Middle School 
Student’ Interest in Science Careers: 

An Exploratory Analysis 

Koszalka, T. 

2002 

2001 Atlanta 
Conference 
Proceedings- 
Research and 
Theory edition, 
vol. 2, 

pp. 338-334 

AECT 

Kids As Airborne Mission Scientists (KaAMS): 
Designing Problem-Based Learning To Inspire 
Kids 

Koszalka, T., 
Kim, Y., and 
Grabowski, B. 

2002 

2001 Atlanta 
Conference 
Proceedings- 
Research and 
Theory edition, 
vol. 1, 
pp. 173-177 

AECT 

Integrating Assessment and Research Strategies 
On a Large Development and Research Project: 
Kids As Airborne Mission Scientists (KaAMS) 

Koszalka, T. and 
Grabowski, B. 

2002 

2002 Lisbon, 
Portugal 
Conference 
Proceedings 

IADIS 
International 
Conference 
WWW/ Internet 
2002 

Helping educators harvest internet resources: 
The development of technology integration 
support tools 

Grabowski, B. and 
Koszalka, T. 

2002 

2002 

Conference 

Proceedings, 

pp. 2117-2121 

Society for 
Information 
Technology and 
Teacher Education 

Relationships between the use of web resources 
and student interests in science: Support for 
technology integration decision-making 

Koszalka, T. 


AECT = Association of Educational Communication Technology; CoSN = Consortium of School Networking; 
1ADIS = International Association for the Development of Information Society; INET = Internet Society; 

LTP = NASA Learning Technologies Project; P-AECT = Pennsylvania AECT 
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WORKSHOPS 


Date 

Type 

Location 

Title 

Facilitators 

Mar-97 


NASA 

What Works and What Doesn't: NASA; Dryden 
Cooperative Agreement Development Teams 

Grabowski, McCarthy, 
Hernandez, and Koszalka 

Jul-97 

Workshop 

Langley NEWEST 

Web Enhanced Learning Environment Strategies 

Grabowski and Koszalka 

Aug-97 

Workshop 

Dryden Teacher 
Ambassadors 

Web Enhanced Learning Environment Strategies 

Grabowski and Koszalka 

Aug-97 

Graduate class 

Penn State Univ. 

Internet in the Classroom 

Peck 

Mar-98 

Workshop 

Keystone Central 
School, PA 

Web Enhanced Learning Environment Strategies 

Grabowski and Koszalka 

Mar-98 

Graduate class 

Penn State Univ. 

Designing WELES for Science, Mathematics, and 
Technology Educators of Native Americans 

Koszalka 

Mar-98 

Workshop 

Thailand 

Web Enhanced Learning Environment Strategies 

Grabowski 

Jun-98 

Graduate class 

Penn State Univ. 

Internet in the Classroom 

Grabowski 

Aug-98 

Graduate class 

Penn State Univ. 

Internet in the Classroom 

Grabowski 

Oct-98 

Workshop 

Keppel, CA 

Web Enhanced Learning Environment Strategies 

Grabowski 

Nov-98 

Workshop 

Altoona, PA 

Web Enhanced Learning Environment 

Grabowski and Koszalka 

Dec-98 

Workshop 

Lewisburg, PA 

Web Enhanced Learning Environment Strategies 

Grabowski and Koszalka 

Dec-98 

Workshop 

Altoona, PA 

Web Enhanced Learning Environment Strategies 

Grabowski and Koszalka 

Jan-99 

Graduate class 

Penn State Univ. 

Internet in the Classroom 

Grabowski 

Mar-99 

Workshop 

Dole Middle 
School, HI 

Integrating Web Resources into your Classroom 
using - WELES 

Grabowski and Koszalka 

Jul-99 

Workshop 

Blacklick Valley, 
PA 

Web Enhanced Learning Environment Strategies 

Koszalka 

Sep-99 

Graduate class 

Penn State Univ. 

Internet in the Classroom 

Koszalka 

Jan-00 

Graduate class 

Penn State Univ. 

Internet in the Classroom 

Koszalka 
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PROJECT PUBLICATIONS 


Year 

Type 

Title 

Authorship 

1997 


The world wide web as a medium of 
instruction: What works and what doesn't 

McCarthy, Grabowski, Hernandez, 
Koszalka, and Duke 

1997 


Web Enhanced Learning Environment 
Strategies 

Grabowski, Koszalka, and 
McCarthy 

1997 

Teaching Guide, versions 1-2 

WELES teaching guide 

Grabowski, Koszalka, and 
McCarthy 

1997 fa 

Penn State INSYS Program 
Newsletter 

PSU and NASA Dryden Join Forces 

Koszalka 

1998 fa 

Penn State INSYS Program 
Newsletter 

WELES showcased at International 
Conference 

Koszalka 

1998 sp 

Penn State INSYS Program 
Newsletter 

PSU & NASA Dryden Flight Research Center: 
Research with Local Teachers 

Koszalka 

1998 

Handbook, versions 4-8 

Web Enhanced Learning Environment 
Strategies Handbooks and Reflection Tool 

Grabowski, Koszalka, and 
McCarthy 

1998 

Lesson Planner, version 1 

Web Enhanced Learning Environment 
Strategies Lesson Planner 

Grabowski, Koszalka, and 
McCarthy 

1998 

Teaching Guide, versions 3^1 

WELES teaching guide 

Grabowski, Koszalka, and 
McCarthy 

1998 

NASA TP 1998 206547 

Web-based Instruction and Learning: 
Analysis and Needs Assessment 

McCarthy and Grabowski 

1998 

INET Conference Proceedings 

Web-based Instruction and Learning: 
Analysis and Needs Assessment 

McCarthy, Grabowski, and 
Koszalka 

1998 

INET Conference Proceedings 

Web-Enhanced Learning Environment 
Strategies for Classroom Teachers 

Grabowski, Koszalka, and 
McCarthy 

1999 

Handbook, versions 9-11 

Web Enhanced Learning Environment 
Strategies Handbooks and Reflection Tool 

Grabowski, Koszalka, and 
McCarthy 

1999 

Lesson Planner, version 2 

Web Enhanced Learning Environment 
Strategies Lesson Planner 

Grabowski, Koszalka, and 
McCarthy 

1999 

Teaching Guide, versions 5-7 

WELES teaching guide 

Grabowski, Koszalka, and 
McCarthy 

1999 

Journal of Educational and 
Information Technologies 

Sharing lesson Plans Over the World Wide 
Web: Important Components 

Koszalka, Breman, and Moore 

1999 

Research Report 

WELES Workshop Summary: 

Dole Middle School — March 1999 

Koszalka, Grabowski, and 
McCarthy 

1999 

Research Report 

WELES Workshop Summary: Lancaster CA, 
Altoona PA, Lewisburg PA — Oct.-Dec. 1998 

Grabowski, Koszalka, and 
McCarthy 

1999 

Research Report 

WELES Workshop Summary: Keystone 
Central School District — March 1998 

Koszalka, Grabowski, and 
McCarthy 

1999 

Research Report 

WELES Workshop Summary: Dryden 

Grabowski, Koszalka, and 
McCarthy 

1999 

Research Report 

WELES Workshop Summary: Langley 

Grabowski, Koszalka, and 
McCarthy 

1999 

Research Report 

Change History and Teacher Exposure to 
WELES Report 

Grabowski, Koszalka, and 
McCarthy 
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PROJECT PUBLICATIONS, CONTINUED 


(Published Refereed Journal Publications) 

Year 

Type 

Title 

Authorship 

2002 

American Educational 
Research Association, 
New Orleans, LA. 
ERIC #IR02 1174 

Designing problem-based learning lesson plan 
structures for middle school teachers: 

Using NASA missions to inspire Kids as 
Airborne Mission Scientists (KaAMS) 

Koszalka, T., 
Grabowski, B., and 
Kim, Y. 

2002 

American Educational 
Research Association, 
New Orleans, LA. 
ERIC #IR02 1176 

Predictive relationships among science 
classroom resources and middle school student 
interest in science careers: 

An exploratory analysis 

Koszalka, T. 

2003 

Evaluation in Program 
Planning: Special Issue: 
Evaluation of Educational 
Technology: International 
Perspectives, 26(2), 203-213. 

Combining Assessment and Research 
Strategies On Large Educational Technology 
Development Projects: Possibilities 
Exemplified through Kids as Airborne Mission 
Scientists (KaAMS) 

Koszalka, T. and 
Grabowski, B. 

2003 

Journal of Technology 
and Teacher Education, 
11(3), 349-378. 

Reflection Through the 1DPRISM: A Teacher 
Planning Tool to Transform Classrooms into 
Web-Enhanced Learning Environments 

Koszalka, T., 
Grabowski, B., and 
McCarthy, M. 


41 


PUBLICATIONS BY OTHERS ABOUT THE WELES PROJECT 


Year 

Type 

Title 

Authorship 

Apr-97 

The Dryden NASA X-Press 

Conference looks at Web and Education 

McMacken 

Nov-97 

Focal Points: PSU College of 
Education Newsletter 

NASA and the College of Education: A Break- 
through Partnership 

Blaum 

Nov-97 

NASA DFRC News Release 
97-47 

Penn State, NASA form Hi-Tech education 
partnership 

NASA Public Affairs Office 

Dec -97 

Centre Daily Times Newspaper 

The Internet is here to stay: Penn State Insys 
teacher sees huge opportunities in 'net 
resources 

Cheng 
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